Trust in Decision Aids: a Model and Its Training Implications

Decision aids have much in common with other types of automation. For example, they vary in the level of automation that they offer — from data integration, through expert systems that generate decision options, to associate systems that take action unless the user overrides. In principle, “autonomous” systems can be developed that evaluate, choose and act without the user’s knowledge. Automation of decision making has not advanced as far along this spectrum as automation in other fields. Explanations for this “failure” vary. According to some advocates of decision aids, the reason is a sort of irrational technophobia, evidenced by a lack of appropriate trust in the decisions that such aids make. According to skeptics, on the other hand, a good reason for rejecting decision aids altogether is overtrust: the tendency of users to rely on an aid as if it were infallible when they should instead rely on their own judgment. Strangely enough, both camps share a similar concept of trust. In both views, trust is a relatively enduring attitude that a user has toward an aid, akin to love, hate, or faith, rather than a more transient and situation-specific attitude, such as agreement or disagreement. We will propose an alternative, more differentiated conception of trust. It includes the more enduring concept as a special case, but emphasizes instead the specific conditions under which an aid will and will not perform well. According to this alternative approach, the problem of decision aid acceptance is neither undertrust nor overtrust, but inappropriate trust: a failure to understand or properly evaluate the conditions affecting good and bad aid performance. The issue of trust marks an important difference between decision aids and other types of automation. Decision aids are often intended to help users handle uncertainty about a domain. Yet, an obstacle to the effective use of decision aids is uncertainty about the decision aids themselves. Unlike other kinds of automation, therefore, decision aids may transform, but not eliminate, the human task that was to have been automated. Existing training has neglected the issue of uncertainty. It typically focuses on what the user must do to make the aid work, i.e., inputs, outputs, and modes of operation. In doing so, it has inadvertently reinforced the misconception that trust must be a permanent stance, to accept or reject an aid as a whole. More often than not, however, to benefit from a decision aid, the user must learn, or be trained, to recognize and act on uncertainty about the quality of the aid’s recommendations, and to understand how such uncertainty can change from situation to situation. In most cases, this task is not trivial. The domains in which decision aids are introduced tend to be complex; novel situations are likely to arise that were not anticipated by aid designers; the workload and task priorities of users may shift, along with the attention they can devote to the decision aid itself; and by the very nature of uncertainty, even the best decision may on occasion have a bad outcome, or a bad decision a good outcome. It is not easy in this context to acquire an understanding of the aid's decision making processes that will support effective exploitation of the aid. The challenge is perhaps not unlike the one we face in learning